Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feature: Google Cloud Storage support skeleton #513

Merged
merged 17 commits into from
Aug 15, 2022

Conversation

ClSlaid
Copy link
Contributor

@ClSlaid ClSlaid commented Aug 12, 2022

Summary

This PR will complete #7.

  1. try implement GCS support backend prototype
  2. try implement GCS support directory walker prototype

Signed-off-by: ClSlaid cailue@bupt.edu.cn

I hereby agree to the terms of the CLA available at: https://databend.rs/dev/policies/cla/

1. try implement GCS support backend prototype
2. try implement GCS support directory walker prototype

Signed-off-by: ClSlaid <cailue@bupt.edu.cn>
@ClSlaid
Copy link
Contributor Author

ClSlaid commented Aug 12, 2022

This PR is still working in progress, tasks to complete:

  • Add unit tests for the GCS backend
  • Add unit tests for the GCS directory walker
  • Add integrated tests for the GCS service
  • Test on real-world GCS

Add Apache license headers for gcs files

Signed-off-by: ClSlaid <cailue@bupt.edu.cn>
@Xuanwo
Copy link
Member

Xuanwo commented Aug 12, 2022

This PR is still working in progress, tasks to complete:

  • Add unit tests for the GCS backend
  • Add unit tests for the GCS directory walker
  • Add integrated tests for the GCS service
  • Test on real-world GCS

Please create a tracking issue for this. I prefer to merge multiple PRs instead of one.

@ClSlaid ClSlaid changed the title feature: implement Google Cloud Storage support feature: Google Cloud Storage support skeleton Aug 12, 2022
Cargo.toml Outdated Show resolved Hide resolved
src/scheme.rs Outdated Show resolved Hide resolved
src/scheme.rs Show resolved Hide resolved
src/scheme.rs Show resolved Hide resolved
src/services/gcs/backend.rs Outdated Show resolved Hide resolved
src/services/gcs/backend.rs Outdated Show resolved Hide resolved
src/services/gcs/backend.rs Outdated Show resolved Hide resolved
src/services/gcs/backend.rs Outdated Show resolved Hide resolved
src/services/gcs/backend.rs Outdated Show resolved Hide resolved
src/services/gcs/backend.rs Outdated Show resolved Hide resolved
1. canceled unneccesary feature switch `service-gcs`
2. implement from_iter() for `gcs::Backend`
3. *make gcs compile*

Signed-off-by: ClSlaid <cailue@bupt.edu.cn>
@Xuanwo
Copy link
Member

Xuanwo commented Aug 12, 2022

I refactored error parse in #515 and #516, PTAL.

added some document comments in gcs

Signed-off-by: ClSlaid <cailue@bupt.edu.cn>
src/services/gcs/backend.rs Outdated Show resolved Hide resolved
now it should be able to compiled and satisfies clippy

Signed-off-by: ClSlaid <cailue@bupt.edu.cn>
Signed-off-by: ClSlaid <cailue@bupt.edu.cn>
src/services/gcs/backend.rs Outdated Show resolved Hide resolved
now the backend should inherits the default implementation of Accessor

Signed-off-by: ClSlaid <cailue@bupt.edu.cn>
Signed-off-by: ClSlaid <cailue@bupt.edu.cn>
Signed-off-by: ClSlaid <cailue@bupt.edu.cn>
Copy link
Member

@Xuanwo Xuanwo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Mostly LGTM.

I raised some reviews for coding style. PTAL.

src/services/gcs/backend.rs Outdated Show resolved Hide resolved
src/services/gcs/backend.rs Outdated Show resolved Hide resolved
src/services/gcs/backend.rs Outdated Show resolved Hide resolved
src/services/gcs/backend.rs Outdated Show resolved Hide resolved
src/services/gcs/backend.rs Outdated Show resolved Hide resolved
src/services/gcs/dir_stream.rs Show resolved Hide resolved
src/services/gcs/dir_stream.rs Outdated Show resolved Hide resolved
src/services/gcs/dir_stream.rs Show resolved Hide resolved
src/services/gcs/dir_stream.rs Outdated Show resolved Hide resolved
src/services/gcs/dir_stream.rs Outdated Show resolved Hide resolved
Signed-off-by: ClSlaid <cailue@bupt.edu.cn>
@Xuanwo Xuanwo marked this pull request as draft August 15, 2022 08:16
Signed-off-by: ClSlaid <cailue@bupt.edu.cn>
Signed-off-by: ClSlaid <cailue@bupt.edu.cn>
src/services/gcs/backend.rs Outdated Show resolved Hide resolved
src/services/gcs/backend.rs Outdated Show resolved Hide resolved
src/services/gcs/meta.rs Outdated Show resolved Hide resolved
src/services/gcs/meta.rs Outdated Show resolved Hide resolved
@Xuanwo
Copy link
Member

Xuanwo commented Aug 15, 2022

Please mark the conversation as resolved after fixing.

Signed-off-by: ClSlaid <cailue@bupt.edu.cn>
@ClSlaid ClSlaid marked this pull request as ready for review August 15, 2022 11:52
@ClSlaid
Copy link
Contributor Author

ClSlaid commented Aug 15, 2022

@Xuanwo PTAL! ;)

Signed-off-by: ClSlaid <cailue@bupt.edu.cn>
src/services/gcs/backend.rs Outdated Show resolved Hide resolved
src/services/gcs/backend.rs Show resolved Hide resolved
src/services/gcs/backend.rs Outdated Show resolved Hide resolved
src/services/gcs/backend.rs Outdated Show resolved Hide resolved
src/services/gcs/backend.rs Show resolved Hide resolved
src/services/gcs/backend.rs Outdated Show resolved Hide resolved
src/services/gcs/dir_stream.rs Outdated Show resolved Hide resolved
Signed-off-by: ClSlaid <cailue@bupt.edu.cn>
Copy link
Member

@Xuanwo Xuanwo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Mostly LGTM. Thanks for contributing!

@Xuanwo Xuanwo merged commit b0ef4b8 into apache:main Aug 15, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants